The background description provided here is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
Search engines are an integral part of today's world. A key component of a search engine is the collection of search indices that power the search. In the context of a search engine, a search index can be an inverted index that associates keywords or combinations of keywords to documents (e.g., web pages) that contain the keyword or combination of keywords. In order to generate and maintain these search indexes, most search engines use crawlers to identify documents and information within the documents. A traditional crawler requests a document from a content provider and the content provider provides the requested document to the crawler. The crawler then identifies and indexes the keywords and combinations of keywords in the document.
As the world transitions to a mobile-based architecture, the way content providers provide access to their content is changing. User devices can access content using a variety of different mechanisms. For example, user devices can obtain content from a content provider using a native application dedicated to accessing a software application of the content provider or a web browser that accesses the software application using a web browser. Furthermore, content providers may allow access to different content depending on the geographic region of a user device, the type of user device, the time of day, and/or the operating system of the user device. For these and other reasons, crawling has become an increasingly difficult task.
Further, presentation of search results on the user device, which are obtained based on a query received from the user device, may be performed in many ways. The latency and quality of rendering the search results on the user device may differ depending on the manner of presenting the search results and can impact user experience.